Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2615 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.8603. Min Act: -0.0001

Data Index: 568191 (C4 (Web Text))

Max Activating Token Index: 387

Text #1

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.8137. Min Act: -0.0001

Data Index: 900158 (C4 (Web Text))

Max Activating Token Index: 410

Text #2

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.8033. Min Act: -0.0001

Data Index: 339819 (C4 (Web Text))

Max Activating Token Index: 387

Text #3

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.7859. Min Act: -0.0001

Data Index: 400552 (C4 (Web Text))

Max Activating Token Index: 436

Text #4

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.7234. Min Act: -0.0000

Data Index: 1058652 (C4 (Web Text))

Max Activating Token Index: 458

Text #5

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6776. Min Act: -0.0000

Data Index: 367770 (C4 (Web Text))

Max Activating Token Index: 900

Text #6

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6541. Min Act: -0.0001

Data Index: 407951 (C4 (Web Text))

Max Activating Token Index: 497

Text #7

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6501. Min Act: -0.0001

Data Index: 685563 (C4 (Web Text))

Max Activating Token Index: 974

Text #8

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6188. Min Act: -0.0001

Data Index: 549225 (C4 (Web Text))

Max Activating Token Index: 861

Text #9

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6332. Min Act: -0.0001

Data Index: 194357 (C4 (Web Text))

Max Activating Token Index: 745

Text #10

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6688. Min Act: -0.0001

Data Index: 610016 (C4 (Web Text))

Max Activating Token Index: 200

Text #11

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6552. Min Act: -0.0001

Data Index: 749325 (C4 (Web Text))

Max Activating Token Index: 726

Text #12

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6080. Min Act: -0.0001

Data Index: 250500 (C4 (Web Text))

Max Activating Token Index: 1016

Text #13

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6047. Min Act: -0.0000

Data Index: 74357 (C4 (Web Text))

Max Activating Token Index: 66

Text #14

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6194. Min Act: -0.0001

Data Index: 999943 (C4 (Web Text))

Max Activating Token Index: 866

Text #15

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.5884. Min Act: -0.0001

Data Index: 624809 (C4 (Web Text))

Max Activating Token Index: 729

Text #16

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.6169. Min Act: -0.0001

Data Index: 203552 (C4 (Web Text))

Max Activating Token Index: 768

Text #17

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.5231. Min Act: -0.0000

Data Index: 1160503 (C4 (Web Text))

Max Activating Token Index: 992

Text #18

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.5764. Min Act: -0.0001

Data Index: 717671 (C4 (Web Text))

Max Activating Token Index: 259

Text #19

Max Range: 0.8603. Min Range: -0.8603

Max Act: 0.5748. Min Act: -0.0000

Data Index: 1007157 (C4 (Web Text))

Max Activating Token Index: 1006