Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4528 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.3678. Min Act: -0.0000

Data Index: 1133995 (C4 (Web Text))

Max Activating Token Index: 887

Text #1

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.3061. Min Act: -0.0000

Data Index: 783852 (C4 (Web Text))

Max Activating Token Index: 970

Text #2

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.2286. Min Act: -0.0000

Data Index: 217218 (C4 (Web Text))

Max Activating Token Index: 478

Text #3

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.1838. Min Act: -0.0000

Data Index: 1238725 (C4 (Web Text))

Max Activating Token Index: 99

Text #4

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.1956. Min Act: -0.0000

Data Index: 601869 (C4 (Web Text))

Max Activating Token Index: 385

Text #5

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.2176. Min Act: 0.0000

Data Index: 299907 (C4 (Web Text))

Max Activating Token Index: 1006

Text #6

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.1359. Min Act: -0.0000

Data Index: 461059 (C4 (Web Text))

Max Activating Token Index: 681

Text #7

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0921. Min Act: -0.0000

Data Index: 466810 (C4 (Web Text))

Max Activating Token Index: 747

Text #8

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.1012. Min Act: 0.0000

Data Index: 380653 (C4 (Web Text))

Max Activating Token Index: 425

Text #9

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0560. Min Act: -0.0000

Data Index: 711432 (C4 (Web Text))

Max Activating Token Index: 790

Text #10

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0485. Min Act: -0.0000

Data Index: 753048 (C4 (Web Text))

Max Activating Token Index: 513

Text #11

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0402. Min Act: -0.0000

Data Index: 93937 (C4 (Web Text))

Max Activating Token Index: 141

Text #12

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0149. Min Act: -0.0000

Data Index: 1005104 (C4 (Web Text))

Max Activating Token Index: 283

Text #13

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0205. Min Act: -0.0000

Data Index: 769726 (C4 (Web Text))

Max Activating Token Index: 956

Text #14

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0131. Min Act: -0.0000

Data Index: 1033126 (C4 (Web Text))

Max Activating Token Index: 46

Text #15

Max Range: 1.3678. Min Range: -1.3678

Max Act: 0.9395. Min Act: -0.0000

Data Index: 112896 (C4 (Web Text))

Max Activating Token Index: 280

Text #16

Max Range: 1.3678. Min Range: -1.3678

Max Act: 1.0247. Min Act: 0.0000

Data Index: 518750 (C4 (Web Text))

Max Activating Token Index: 417

Text #17

Max Range: 1.3678. Min Range: -1.3678

Max Act: 0.9957. Min Act: -0.0000

Data Index: 485893 (C4 (Web Text))

Max Activating Token Index: 615

Text #18

Max Range: 1.3678. Min Range: -1.3678

Max Act: 0.9033. Min Act: -0.0000

Data Index: 188562 (C4 (Web Text))

Max Activating Token Index: 529

Text #19

Max Range: 1.3678. Min Range: -1.3678

Max Act: 0.9527. Min Act: -0.0000

Data Index: 1999 (C4 (Web Text))

Max Activating Token Index: 50