Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2856 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0894. Min Range: -3.0894

Max Act: 3.0894. Min Act: -0.0001

Data Index: 853136 (C4 (Web Text))

Max Activating Token Index: 57

Text #1

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.7491. Min Act: -0.0001

Data Index: 1339367 (C4 (Web Text))

Max Activating Token Index: 10

Text #2

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.7489. Min Act: -0.0001

Data Index: 509931 (C4 (Web Text))

Max Activating Token Index: 692

Text #3

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.5349. Min Act: -0.0000

Data Index: 418203 (C4 (Web Text))

Max Activating Token Index: 13

Text #4

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.6500. Min Act: -0.0001

Data Index: 1071870 (C4 (Web Text))

Max Activating Token Index: 4

Text #5

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.4002. Min Act: -0.0000

Data Index: 1144491 (C4 (Web Text))

Max Activating Token Index: 18

Text #6

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.6180. Min Act: -0.0001

Data Index: 385183 (C4 (Web Text))

Max Activating Token Index: 800

Text #7

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.3325. Min Act: -0.0001

Data Index: 141033 (C4 (Web Text))

Max Activating Token Index: 755

Text #8

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.1554. Min Act: -0.0001

Data Index: 1224380 (C4 (Web Text))

Max Activating Token Index: 28

Text #9

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.2909. Min Act: -0.0001

Data Index: 1206964 (C4 (Web Text))

Max Activating Token Index: 492

Text #10

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.2770. Min Act: -0.0001

Data Index: 958477 (C4 (Web Text))

Max Activating Token Index: 483

Text #11

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.2729. Min Act: -0.0001

Data Index: 80806 (C4 (Web Text))

Max Activating Token Index: 164

Text #12

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.2530. Min Act: -0.0001

Data Index: 1240950 (C4 (Web Text))

Max Activating Token Index: 34

Text #13

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.1295. Min Act: -0.0001

Data Index: 832236 (C4 (Web Text))

Max Activating Token Index: 15

Text #14

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.2464. Min Act: -0.0001

Data Index: 527505 (C4 (Web Text))

Max Activating Token Index: 288

Text #15

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.2062. Min Act: -0.0001

Data Index: 1189396 (C4 (Web Text))

Max Activating Token Index: 286

Text #16

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.0488. Min Act: -0.0001

Data Index: 1058344 (C4 (Web Text))

Max Activating Token Index: 7

Text #17

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.0564. Min Act: -0.0001

Data Index: 16415 (C4 (Web Text))

Max Activating Token Index: 833

Text #18

Max Range: 3.0894. Min Range: -3.0894

Max Act: 1.8992. Min Act: -0.0001

Data Index: 1028390 (C4 (Web Text))

Max Activating Token Index: 726

Text #19

Max Range: 3.0894. Min Range: -3.0894

Max Act: 2.2423. Min Act: -0.0001

Data Index: 980293 (C4 (Web Text))

Max Activating Token Index: 243