Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1083 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1409. Min Range: -1.1409

Max Act: 1.1409. Min Act: -0.0000

Data Index: 1229345 (C4 (Web Text))

Max Activating Token Index: 58

Text #1

Max Range: 1.1409. Min Range: -1.1409

Max Act: 1.1284. Min Act: -0.0000

Data Index: 835189 (C4 (Web Text))

Max Activating Token Index: 21

Text #2

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.9946. Min Act: -0.0000

Data Index: 297764 (C4 (Web Text))

Max Activating Token Index: 64

Text #3

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.9598. Min Act: -0.0000

Data Index: 473221 (C4 (Web Text))

Max Activating Token Index: 252

Text #4

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.9582. Min Act: -0.0000

Data Index: 976091 (C4 (Web Text))

Max Activating Token Index: 19

Text #5

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.9832. Min Act: -0.0000

Data Index: 416050 (C4 (Web Text))

Max Activating Token Index: 464

Text #6

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8941. Min Act: -0.0000

Data Index: 581989 (C4 (Web Text))

Max Activating Token Index: 819

Text #7

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.9582. Min Act: -0.0000

Data Index: 551616 (C4 (Web Text))

Max Activating Token Index: 757

Text #8

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.9918. Min Act: -0.0000

Data Index: 508334 (C4 (Web Text))

Max Activating Token Index: 934

Text #9

Max Range: 1.1409. Min Range: -1.1409

Max Act: 1.0851. Min Act: 0.0001

Data Index: 265976 (C4 (Web Text))

Max Activating Token Index: 300

Text #10

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8982. Min Act: 0.0001

Data Index: 286660 (C4 (Web Text))

Max Activating Token Index: 653

Text #11

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8602. Min Act: -0.0000

Data Index: 506940 (C4 (Web Text))

Max Activating Token Index: 463

Text #12

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8626. Min Act: 0.0001

Data Index: 741179 (C4 (Web Text))

Max Activating Token Index: 308

Text #13

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8650. Min Act: -0.0000

Data Index: 1065492 (C4 (Web Text))

Max Activating Token Index: 114

Text #14

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8086. Min Act: -0.0000

Data Index: 264109 (C4 (Web Text))

Max Activating Token Index: 391

Text #15

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8501. Min Act: -0.0000

Data Index: 956507 (C4 (Web Text))

Max Activating Token Index: 707

Text #16

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8473. Min Act: -0.0000

Data Index: 904492 (C4 (Web Text))

Max Activating Token Index: 653

Text #17

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8468. Min Act: 0.0001

Data Index: 1042143 (C4 (Web Text))

Max Activating Token Index: 964

Text #18

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8455. Min Act: -0.0000

Data Index: 1305760 (C4 (Web Text))

Max Activating Token Index: 942

Text #19

Max Range: 1.1409. Min Range: -1.1409

Max Act: 0.8117. Min Act: -0.0000

Data Index: 954691 (C4 (Web Text))

Max Activating Token Index: 215