Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 681 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1702. Min Range: -2.1702

Max Act: 2.1680. Min Act: -0.0000

Data Index: 1619293 (Python Code)

Max Activating Token Index: 189

Text #1

Max Range: 2.1702. Min Range: -2.1702

Max Act: 2.1702. Min Act: -0.0000

Data Index: 1370000 (Python Code)

Max Activating Token Index: 373

Text #2

Max Range: 2.1702. Min Range: -2.1702

Max Act: 2.1570. Min Act: -0.0000

Data Index: 1382592 (Python Code)

Max Activating Token Index: 43

Text #3

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.8620. Min Act: -0.0000

Data Index: 1517633 (Python Code)

Max Activating Token Index: 885

Text #4

Max Range: 2.1702. Min Range: -2.1702

Max Act: 2.0425. Min Act: -0.0000

Data Index: 1455912 (Python Code)

Max Activating Token Index: 82

Text #5

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.8521. Min Act: -0.0000

Data Index: 1445961 (Python Code)

Max Activating Token Index: 608

Text #6

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.8409. Min Act: -0.0000

Data Index: 1474446 (Python Code)

Max Activating Token Index: 100

Text #7

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.9212. Min Act: -0.0000

Data Index: 1418585 (Python Code)

Max Activating Token Index: 414

Text #8

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.9322. Min Act: -0.0000

Data Index: 1420087 (Python Code)

Max Activating Token Index: 271

Text #9

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.9089. Min Act: -0.0000

Data Index: 1573628 (Python Code)

Max Activating Token Index: 25

Text #10

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.7510. Min Act: -0.0000

Data Index: 1566529 (Python Code)

Max Activating Token Index: 786

Text #11

Max Range: 2.1702. Min Range: -2.1702

Max Act: 2.0280. Min Act: -0.0000

Data Index: 1539940 (Python Code)

Max Activating Token Index: 905

Text #12

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.7270. Min Act: -0.0000

Data Index: 1414460 (Python Code)

Max Activating Token Index: 792

Text #13

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.8074. Min Act: -0.0000

Data Index: 1469909 (Python Code)

Max Activating Token Index: 483

Text #14

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.7866. Min Act: -0.0000

Data Index: 1511927 (Python Code)

Max Activating Token Index: 143

Text #15

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.6731. Min Act: -0.0000

Data Index: 1422025 (Python Code)

Max Activating Token Index: 321

Text #16

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.7632. Min Act: -0.0000

Data Index: 1419697 (Python Code)

Max Activating Token Index: 919

Text #17

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.7462. Min Act: -0.0000

Data Index: 1469321 (Python Code)

Max Activating Token Index: 475

Text #18

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.7541. Min Act: -0.0000

Data Index: 1639277 (Python Code)

Max Activating Token Index: 463

Text #19

Max Range: 2.1702. Min Range: -2.1702

Max Act: 1.7477. Min Act: -0.0000

Data Index: 1628009 (Python Code)

Max Activating Token Index: 89