Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4003 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.9162. Min Act: -0.0000

Data Index: 522484 (C4 (Web Text))

Max Activating Token Index: 285

Text #1

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.9416. Min Act: -0.0001

Data Index: 581353 (C4 (Web Text))

Max Activating Token Index: 112

Text #2

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.8336. Min Act: -0.0000

Data Index: 541907 (C4 (Web Text))

Max Activating Token Index: 436

Text #3

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.8821. Min Act: -0.0000

Data Index: 1024257 (C4 (Web Text))

Max Activating Token Index: 590

Text #4

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7990. Min Act: -0.0000

Data Index: 1342007 (C4 (Web Text))

Max Activating Token Index: 685

Text #5

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.8186. Min Act: -0.0000

Data Index: 603120 (C4 (Web Text))

Max Activating Token Index: 812

Text #6

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7934. Min Act: -0.0000

Data Index: 243726 (C4 (Web Text))

Max Activating Token Index: 95

Text #7

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7929. Min Act: -0.0000

Data Index: 329626 (C4 (Web Text))

Max Activating Token Index: 419

Text #8

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.8183. Min Act: -0.0001

Data Index: 238790 (C4 (Web Text))

Max Activating Token Index: 426

Text #9

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7908. Min Act: -0.0001

Data Index: 1077833 (C4 (Web Text))

Max Activating Token Index: 769

Text #10

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7623. Min Act: -0.0000

Data Index: 1348444 (C4 (Web Text))

Max Activating Token Index: 785

Text #11

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7583. Min Act: -0.0000

Data Index: 237916 (C4 (Web Text))

Max Activating Token Index: 702

Text #12

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.8090. Min Act: -0.0000

Data Index: 160266 (C4 (Web Text))

Max Activating Token Index: 846

Text #13

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7725. Min Act: -0.0000

Data Index: 207953 (C4 (Web Text))

Max Activating Token Index: 460

Text #14

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7871. Min Act: -0.0000

Data Index: 746039 (C4 (Web Text))

Max Activating Token Index: 201

Text #15

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7422. Min Act: -0.0000

Data Index: 353402 (C4 (Web Text))

Max Activating Token Index: 973

Text #16

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7357. Min Act: -0.0000

Data Index: 632031 (C4 (Web Text))

Max Activating Token Index: 777

Text #17

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.6907. Min Act: -0.0000

Data Index: 380566 (C4 (Web Text))

Max Activating Token Index: 654

Text #18

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7298. Min Act: -0.0000

Data Index: 90016 (C4 (Web Text))

Max Activating Token Index: 646

Text #19

Max Range: 0.9416. Min Range: -0.9416

Max Act: 0.7520. Min Act: -0.0000

Data Index: 891483 (C4 (Web Text))

Max Activating Token Index: 756