Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 519 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8841. Min Act: -0.0000

Data Index: 697460 (C4 (Web Text))

Max Activating Token Index: 415

Text #1

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8528. Min Act: -0.0000

Data Index: 705406 (C4 (Web Text))

Max Activating Token Index: 43

Text #2

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8783. Min Act: -0.0000

Data Index: 1283479 (C4 (Web Text))

Max Activating Token Index: 397

Text #3

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8316. Min Act: -0.0000

Data Index: 231287 (C4 (Web Text))

Max Activating Token Index: 63

Text #4

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8528. Min Act: -0.0000

Data Index: 1358905 (C4 (Web Text))

Max Activating Token Index: 529

Text #5

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8098. Min Act: -0.0000

Data Index: 255709 (C4 (Web Text))

Max Activating Token Index: 666

Text #6

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7737. Min Act: -0.0000

Data Index: 1013157 (C4 (Web Text))

Max Activating Token Index: 273

Text #7

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7981. Min Act: -0.0000

Data Index: 10090 (C4 (Web Text))

Max Activating Token Index: 804

Text #8

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8196. Min Act: -0.0000

Data Index: 282384 (C4 (Web Text))

Max Activating Token Index: 223

Text #9

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8430. Min Act: -0.0000

Data Index: 699397 (C4 (Web Text))

Max Activating Token Index: 379

Text #10

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8110. Min Act: -0.0000

Data Index: 819719 (C4 (Web Text))

Max Activating Token Index: 49

Text #11

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8386. Min Act: -0.0000

Data Index: 229661 (C4 (Web Text))

Max Activating Token Index: 548

Text #12

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7793. Min Act: -0.0000

Data Index: 979898 (C4 (Web Text))

Max Activating Token Index: 872

Text #13

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7773. Min Act: -0.0000

Data Index: 10020 (C4 (Web Text))

Max Activating Token Index: 56

Text #14

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8048. Min Act: -0.0000

Data Index: 1296428 (C4 (Web Text))

Max Activating Token Index: 204

Text #15

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.8041. Min Act: -0.0000

Data Index: 716299 (C4 (Web Text))

Max Activating Token Index: 401

Text #16

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7881. Min Act: -0.0000

Data Index: 445469 (C4 (Web Text))

Max Activating Token Index: 234

Text #17

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7835. Min Act: -0.0000

Data Index: 939916 (C4 (Web Text))

Max Activating Token Index: 41

Text #18

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7848. Min Act: -0.0000

Data Index: 440129 (C4 (Web Text))

Max Activating Token Index: 330

Text #19

Max Range: 0.8841. Min Range: -0.8841

Max Act: 0.7142. Min Act: -0.0000

Data Index: 1058112 (C4 (Web Text))

Max Activating Token Index: 159