pkg/curlx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

初始化项目

Browse Source
This commit is contained in:
Yun
2023-08-06 01:17:27 +08:00
parent fca2168f8a
commit ee1de00721
7 changed files with 625 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -86
View File
@@ -1,92 +1,10 @@
# curlx
# 项目介绍
这是服务端请求的封装
## Getting started
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
## Add your files
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
```
cd existing_repo
git remote add origin https://code.yun.ink/open/curlx.git
git branch -M master
git push -uf origin master
```
## Integrate with your tools
- [ ] [Set up project integrations](https://code.yun.ink/open/curlx/-/settings/integrations)
## Collaborate with your team
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
## Test and Deploy
Use the built-in continuous integration in GitLab.
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
***
# Editing this README
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
## Name
Choose a self-explaining name for your project.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
## License
For open source projects, say how it is licensed.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
curl_text.go
+5
View File
@@ -0,0 +1,5 @@
package curlx
import "testing"
func TestCurlx(t *testing.T) {}
curlx.go
+253
View File
@@ -0,0 +1,253 @@
package curlx
import (
"bufio"
"compress/gzip"
"context"
"fmt"
"io"
"net"
"net/http"
"time"
"golang.org/x/net/proxy"
)
/**
* Author: Yun
* Date: 2023年7月12日11:35:01
*/
type dataType string
const (
DataTypeForm dataType = "form"
DataTypeJson dataType = "json"
DataTypeXml dataType = "xml"
DataTypeEncode dataType = "encode"
DataTypeText dataType = "text"
)
type method string
const (
MethodGet method = "GET"
MethodPost method = "POST"
)
type CurlParams struct {
Url string
Method method // GET/POST
Params interface{}
Headers map[string]interface{}
Cookies interface{}
DataType dataType // FORM,JSON,XML
}
var (
defaultTimeOut = 180 // 默认(最长超时时间)
// 默认的transport
transport http.Transport = http.Transport{
// Dial: func(netw, addr string) (net.Conn, error) {
// // 这里指定域名访问的IP
// // if addr == "api.hk.blueoceanpay.com:443" {
// // addr = "47.56.200.21:443"
// // }
// conn, err := net.DialTimeout(netw, addr, time.Second*time.Duration(timeOut)) // 设置建立连接超时
// if err != nil {
// return nil, err
// }
// // conn.RemoteAddr().String()
// conn.SetDeadline(time.Now().Add(time.Second * time.Duration(timeOut))) // 设置发送接收数据超时
// return conn, nil
// },
// ResponseHeaderTimeout: time.Second * time.Duration(defaultTimeOut), // 响应超时
DisableKeepAlives: false, // 短连接（默认是使用长连接，连接过多时会造成服务器拒绝服务问题）
MaxIdleConns: 0, // 所有host的连接池最大连接数量，默认无穷大
MaxIdleConnsPerHost: 5, // 每个host的连接池最大空闲连接收，默认2
MaxConnsPerHost: 0, // 每个host的最大连接数量
IdleConnTimeout: time.Second * 2, // 空闲连接超时关闭的时间
}
// client = &http.Client{}
)
type DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
type curlx struct {
transport *http.Transport
}
func NewCurlx() *curlx {
return &curlx{
transport: &transport,
}
}
/**
* 使用Socks5代理
*/
func (c *curlx) WithSocks5(address string) error {
baseDialer := &net.Dialer{
Timeout: 180 * time.Second,
KeepAlive: 180 * time.Second,
}
dialSocksProxy, err := proxy.SOCKS5("tcp", address, nil, baseDialer)
if err != nil {
fmt.Println("proxy.SOCKS5 err", err)
return err
}
dialContext := (baseDialer).DialContext
if contextDialer, ok := dialSocksProxy.(proxy.ContextDialer); ok {
dialContext = contextDialer.DialContext
}
c.transport.DialContext = dialContext
return nil
}
/**
* 不校验HTTPS证书
*/
func (c *curlx) WithInsecureSkipVerify() {
c.transport.TLSClientConfig.InsecureSkipVerify = true
}
/**
* 设置超时时间,单位秒
*/
func (c *curlx) WithTimeout(timeout int) {
c.transport.ResponseHeaderTimeout = time.Second * time.Duration(timeout)
}
/**
* 简单请求
*/
func (c *curlx) Send(ctx context.Context, p *CurlParams) (res string, httpcode int, err error) {
response, err := c.sendExec(ctx, p)
if err != nil {
return "", -1, err
}
defer response.Body.Close() // 处理完关闭
// stdout := os.Stdout // 将结果定位到标准输出，也可以直接打印出来，或定位到其他地方进行相应处理
// _, err = io.Copy(stdout, response.Body) // 将第二个参数拷贝到第一个参数，直到第二参数到达EOF或发生错误，返回拷贝的值
status := response.StatusCode // 获取状态码，正常是200
var body []byte
switch response.Header.Get("Content-Encoding") {
case "gzip":
reader, err := gzip.NewReader(response.Body)
if err != nil {
return "", status, err
}
for {
buf := make([]byte, 1024)
n, err := reader.Read(buf)
if err != nil && err != io.EOF {
panic(err)
}
if n == 0 {
break
}
body = append(body, buf...)
}
default:
body, _ = io.ReadAll(response.Body)
}
return string(body), status, nil
}
/**
* 执行发送
* 注意：外部使用需要加这一句 defer response.Body.Close()
*/
func (c *curlx) sendExec(ctx context.Context, p *CurlParams) (resp *http.Response, err error) {
client := &http.Client{
Timeout: time.Second * time.Duration(defaultTimeOut), // 设置该条连接的超时
Transport: c.transport, //
}
err = p.parseMethod()
if err != nil {
return nil, err
}
// 判断和处理url
err = p.parseUrl()
if err != nil {
return nil, err
}
// 处理参数
reqParams, err := p.parseParams()
if err != nil {
return nil, err
}
// 初始化句柄
request, err := http.NewRequest( // 提交请求 用指定的方法
string(p.Method),
p.Url,
reqParams,
)
if err != nil {
return nil, err
}
// 这里指定要访问的HOST,到时候服务器获取主机是获取到这个
// request.Host = "api.hk.blueoceantech.co"
// 设置上下文控制
request = request.WithContext(ctx)
// 处理请求头
p.parseHeaders(request)
// 处理Cookies
p.parseCookies(request)
// 发起请求
response, err := client.Do(request)
if err != nil {
return nil, err
}
return response, nil
}
/**
* 流式请求
*/
func (c *curlx) SendChan(ctx context.Context, p *CurlParams) (<-chan string, error) {
data := make(chan string, 1000)
go func() {
defer close(data)
ctx, cancel := context.WithTimeout(context.Background(), time.Minute*30)
defer cancel()
response, err := c.sendExec(ctx, p)
if err != nil {
return
}
defer response.Body.Close() // 处理完关闭
scanner := bufio.NewScanner(response.Body)
for scanner.Scan() {
text := scanner.Text()
if text == "" {
continue
}
// 30min超时
select {
case <-ctx.Done():
return
case data <- text:
}
}
}()
return data, nil
}
go.mod
+13
View File
@@ -0,0 +1,13 @@
module code.yun.ink/open/curlx
go 1.19
require (
github.com/tidwall/gjson v1.15.0
golang.org/x/net v0.14.0
)
require (
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
)
go.sum
+8
View File
@@ -0,0 +1,8 @@
github.com/tidwall/gjson v1.15.0 h1:5n/pM+v3r5ujuNl4YLZLsQ+UE5jlkLVm7jMzT5Mpolw=
github.com/tidwall/gjson v1.15.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14=
golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI=
request.go
+218
View File
@@ -0,0 +1,218 @@
package curlx
import (
"bytes"
"encoding/json"
"encoding/xml"
"errors"
"io"
"net/http"
"net/url"
"strconv"
"strings"
)
/**
* 处理请求类型
*/
func (p *CurlParams) parseMethod() error {
if p.Method == "" {
return errors.New("请求类型不能为空")
}
return nil
}
/**
* 处理URL
*/
func (p *CurlParams) parseUrl() error {
_, err := url.Parse(p.Url)
if err != nil {
return err
}
return nil
}
/**
* 处理请求头Header
*/
func (p *CurlParams) parseHeaders(r *http.Request) {
if p.Headers != nil {
if r.Header.Get("User-Agent") == "" {
r.Header.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0 Send By Golang")
}
for k, v := range p.Headers {
if vv, ok := v.(string); ok {
r.Header.Set(k, vv)
continue
}
if vv, ok := v.([]string); ok {
for _, vvv := range vv {
r.Header.Add(k, vvv)
}
}
}
}
}
/**
* 处理请求参数
*/
func (p *CurlParams) parseParams() (str io.Reader, err error) {
err = nil
// 初始化(如未初始化)
if p.Headers == nil {
p.Headers = make(map[string]interface{})
}
if p.Params != nil {
if p.DataType == DataTypeJson {
// 判断是否存在
if _, ok := p.Headers["Content-Type"]; !ok {
p.Headers["Content-Type"] = "application/json"
}
strParam, ok := p.Params.(string)
if ok {
return bytes.NewReader([]byte(strParam)), nil
}
b, err := json.Marshal(p.Params)
if err == nil {
return bytes.NewReader(b), nil
}
} else if p.DataType == DataTypeXml {
if _, ok := p.Headers["Content-Type"]; !ok {
p.Headers["Content-Type"] = "application/xml"
}
var string_data string
if value, ok := p.Params.(string); ok {
string_data = string(value)
} else {
var by []byte
by, err = xml.Marshal(p.Params)
if err != nil {
return
}
string_data = string(by)
}
return strings.NewReader(string_data), nil
// switch p.Params.(type) {
// case map[string]string:
// // 请求参数转换成xml结构
// b, err := goutils.Map2XML(p.Params.(map[string]string))
// if err == nil {
// return bytes.NewBuffer(b)
// }
// default:
// b, err := xml.Marshal(p.Params)
// if err == nil {
// return bytes.NewBuffer(b)
// }
// }
} else if p.DataType == DataTypeText {
if _, ok := p.Headers["Content-Type"]; !ok {
p.Headers["Content-Type"] = "text/plain"
}
var string_data string
if value, ok := p.Params.(string); ok {
string_data = string(value)
} else {
err = errors.New("TEXT类型的参数仅支持字符串")
return
}
return strings.NewReader(string_data), nil
} else {
// FORM,""
if _, ok := p.Headers["Content-Type"]; !ok {
p.Headers["Content-Type"] = "application/x-www-form-urlencoded"
}
// 判断需要map[string]interface{}类型
paramValue, ok := p.Params.(map[string]interface{})
if !ok {
return strings.NewReader(""), errors.New("参数需map[string]interface{}")
}
values := url.Values{}
for k, v := range paramValue {
// 字符串
if v_string, ok := v.(string); ok {
values.Set(k, v_string)
}
// 字符串切片
if vv, ok := v.([]string); ok {
for _, vvv := range vv {
values.Add(k+"[]", vvv)
}
}
// int转string
if v_int, ok := v.(int); ok {
values.Set(k, strconv.Itoa(v_int))
}
// int64转string
if v_int64, ok := v.(int64); ok {
values.Set(k, strconv.FormatInt(v_int64, 10))
}
// float32转string
if v_float32, ok := v.(float32); ok {
values.Set(k, strconv.FormatFloat(float64(v_float32), 'f', -1, 32))
}
// float64转string
if v_float64, ok := v.(float64); ok {
values.Set(k, strconv.FormatFloat(v_float64, 'f', -1, 64))
}
}
return strings.NewReader(values.Encode()), nil
}
}
return
}
/**
* 处理Cookie
*/
func (p *CurlParams) parseCookies(r *http.Request) {
switch p.Cookies.(type) {
case string:
cookies := p.Cookies.(string)
r.Header.Add("Cookie", cookies)
case map[string]string:
cookies := p.Cookies.(map[string]string)
for k, v := range cookies {
r.AddCookie(&http.Cookie{
Name: k,
Value: v,
})
}
case []*http.Cookie:
cookies := p.Cookies.([]*http.Cookie)
for _, cookie := range cookies {
r.AddCookie(cookie)
}
}
}
// func (r *Request) parseQuery() {
// switch r.opts.Query.(type) {
// case string:
// str := r.opts.Query.(string)
// r.req.URL.RawQuery = str
// case map[string]interface{}:
// q := r.req.URL.Query()
// for k, v := range r.opts.Query.(map[string]interface{}) {
// if vv, ok := v.(string); ok {
// q.Set(k, vv)
// continue
// }
// if vv, ok := v.([]string); ok {
// for _, vvv := range vv {
// q.Add(k, vvv)
// }
// }
// }
// r.req.URL.RawQuery = q.Encode()
// }
// }
resopnse.go
+124
View File
@@ -0,0 +1,124 @@
package curlx
import (
"net"
"net/http"
"strings"
"github.com/tidwall/gjson"
)
// Response response object
type Response struct {
resp *http.Response
req *http.Request
body []byte
err error
}
// ResponseBody response body
type ResponseBody []byte
// String fmt outout
func (r ResponseBody) String() string {
return string(r)
}
// Read get slice of response body
func (r ResponseBody) Read(length int) []byte {
if length > len(r) {
length = len(r)
}
return r[:length]
}
// GetContents format response body as string
func (r ResponseBody) GetContents() string {
return string(r)
}
// GetRequest get request object
func (r *Response) GetRequest() *http.Request {
return r.req
}
// GetBody parse response body
func (r *Response) GetBody() (ResponseBody, error) {
return ResponseBody(r.body), r.err
}
// GetParsedBody parse response body with gjson
func (r *Response) GetParsedBody() (*gjson.Result, error) {
pb := gjson.ParseBytes(r.body)
return &pb, nil
}
// GetStatusCode get response status code
func (r *Response) GetStatusCode() int {
return r.resp.StatusCode
}
// GetReasonPhrase get response reason phrase
func (r *Response) GetReasonPhrase() string {
status := r.resp.Status
arr := strings.Split(status, " ")
return arr[1]
}
// IsTimeout get if request is timeout
func (r *Response) IsTimeout() bool {
if r.err == nil {
return false
}
netErr, ok := r.err.(net.Error)
if !ok {
return false
}
if netErr.Timeout() {
return true
}
return false
}
// GetHeaders get response headers
func (r *Response) GetHeaders() map[string][]string {
return r.resp.Header
}
// GetHeader get response header
func (r *Response) GetHeader(name string) []string {
headers := r.GetHeaders()
for k, v := range headers {
if strings.ToLower(name) == strings.ToLower(k) {
return v
}
}
return nil
}
// GetHeaderLine get a single response header
func (r *Response) GetHeaderLine(name string) string {
header := r.GetHeader(name)
if len(header) > 0 {
return header[0]
}
return ""
}
// HasHeader get if header exsits in response headers
func (r *Response) HasHeader(name string) bool {
headers := r.GetHeaders()
for k := range headers {
if strings.ToLower(name) == strings.ToLower(k) {
return true
}
}
return false
}