Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 796 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1394. Min Act: -0.0000

Data Index: 264065 (The Pile)

Max Activating Token Index: 787

Text #1

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1409. Min Act: -0.0000

Data Index: 1402953 (The Pile)

Max Activating Token Index: 636

Text #2

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1255. Min Act: -0.0000

Data Index: 1428546 (The Pile)

Max Activating Token Index: 752

Text #3

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1203. Min Act: -0.0000

Data Index: 1618306 (The Pile)

Max Activating Token Index: 406

Text #4

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1274. Min Act: -0.0000

Data Index: 243440 (The Pile)

Max Activating Token Index: 188

Text #5

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1176. Min Act: -0.0000

Data Index: 1657809 (The Pile)

Max Activating Token Index: 954

Text #6

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1022. Min Act: -0.0000

Data Index: 1184521 (The Pile)

Max Activating Token Index: 723

Text #7

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1214. Min Act: -0.0000

Data Index: 799628 (The Pile)

Max Activating Token Index: 697

Text #8

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1119. Min Act: -0.0000

Data Index: 1157555 (The Pile)

Max Activating Token Index: 345

Text #9

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1092. Min Act: -0.0000

Data Index: 1991498 (The Pile)

Max Activating Token Index: 333

Text #10

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.0888. Min Act: -0.0000

Data Index: 717304 (The Pile)

Max Activating Token Index: 535

Text #11

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1088. Min Act: -0.0000

Data Index: 1317057 (The Pile)

Max Activating Token Index: 281

Text #12

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1092. Min Act: -0.0000

Data Index: 1918539 (The Pile)

Max Activating Token Index: 989

Text #13

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1058. Min Act: -0.0000

Data Index: 685701 (The Pile)

Max Activating Token Index: 301

Text #14

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1062. Min Act: -0.0000

Data Index: 795171 (The Pile)

Max Activating Token Index: 332

Text #15

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.0837. Min Act: -0.0000

Data Index: 1970587 (The Pile)

Max Activating Token Index: 140

Text #16

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1063. Min Act: -0.0000

Data Index: 976823 (The Pile)

Max Activating Token Index: 815

Text #17

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.0994. Min Act: -0.0000

Data Index: 1246212 (The Pile)

Max Activating Token Index: 387

Text #18

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1070. Min Act: -0.0000

Data Index: 127910 (The Pile)

Max Activating Token Index: 980

Text #19

Max Range: 0.1409. Min Range: -0.1409

Max Act: 0.1099. Min Act: -0.0000

Data Index: 762646 (The Pile)

Max Activating Token Index: 778